Messaging systems are used to transmit data to consumers. In messaging systems, data intended for a consumer is placed on a message queue whereby the messaging system stores the data until the consumer retrieves the data. The messaging systems can be complex, involving multiple queues, multiple applications, routing of messages and intermediary applications that may alter the content of the message.
The messaging systems are not normally aware of the content of the message, nor the type and structure of messages expected at a particular queue. Messages may contain invalid data when created, or be altered on route to become invalid. It is also possible for messages to be delivered to the wrong queue.
Messaging consumers (also referred to as receivers or listening applications) service queues in the messaging system and expect messages to arrive in a certain format. If messages have been corrupted or delivered to the wrong queue, these poison messages can cause errors in the consumer. Message consumers are unable to handle some of their received messages, because they cannot cope with messages of certain received types or formats.
It is possible to include complex code in the consumer to validate the structure of each message, but this is error prone and expensive to maintain.
In conventional solutions, a message which cannot be handled by the consumer is put back on the input queue and is then retried, rejected, retried and rejected, until a failure count is reached. The message is then transferred to a ‘dead letter queue’ for clean up. This is acceptable for some consumers since the message retrievals and replacements that precede the message being placed on the dead letter queue are typically handled quickly and efficiently between the consumer and queue manager.
Placing the onus on the consumer to determine the validity of the message increases the quantity of application logic, as an application developer has to consider not only the cases where the message content is valid, but also the invalid case. This increases the complexity and development time of such consumer messaging applications. In systems where the message is rolled back to the messaging system, the consumer also needs to ensure that all other work has been done within a transaction and is rolled-back, or otherwise undone. Repeated-rolled back messages may be redelivered to the consumer, and could cause the messaging system to abort processing of the queue. Therefore, application development time is increased to handle invalid messages and/or application execution time is increased during the processing of invalid messages.
This conventional approach is not acceptable for all customers. Some customers (such as banks, airlines and other large corporations) rely on messaging systems to handle very large numbers of business-critical messages and they demand highly robust message delivery and processing. Their message consuming applications tend to be written with a highly cautious approach to problem messages, taking exceptional actions for any application exception that is triggered by identification of a problem message. Such customers often require assured message delivery but they also often want a guaranteed ‘high quality’ of messages reaching their consumer applications, i.e. they want to see minimal exceptions.
Typical messaging middleware does not provide guarantees that the messages will be acceptable to the consumer. It cannot do so because the messaging paradigm tends to implement the middleware functions without consideration of the particular requirements of individual applications, while the applications are shielded from the complexity of the middleware. This either leaves the consumers with sole responsibility for handling problem messages or relies on failure counts as mentioned above.